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METHOD oTCOMBINING DATA ENTRY OraANDWRITTEN 
SYMBOLS WITH DISPLAYED CHARACTER DATA 

FIELD 

The present invention relates to a method for combining data entry produced with 
5 a stylus on a sensing surface such as a computer touch screen or digitising tablet, with display of 
the character data corresponding to each handwritten symbol. Handwriting recognition software 
is used to produce the character data corresponding to each symbol. 

BACKGROUND 

10 Systems with handwriting recognition include electronic notebooks and personal 

digital assistants (PDAs), which are portable computers incorporating a touch screen graphics 
display; and also non-portable computer workstations equipped with a digitising tablet and 
graphics display. Both types of systems have a pen input function when the user draws or writes 
with a stylus on the surface of the touch screen or digitising tablet. For handwritten data entry, 

15 such systems utilize a graphical user interface (GUI) presenting two spatially separate visual 
fields on the graphics display: first, a field where text characters are to be inserted by a text 
editing software program into a document (display field), usually showing a cursor to indicate 
the point of insertion for character data; and second, one or more fields (entry fields), where the 
user draws with the stylus to enter handwritten data. 
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After recognition and conversion of the handwritten data, the resulting character 
data appear in the display field at the point of insertion indicated by the cursor. In a typical 
design, not only are the entry and display fields spatially separate, but also the position, size, 
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location, and other fea^fc of the character data bear Little relatioi^^he appearance of the 
original handwritten input. 



When the stylus is moved outside of an entry field, it typically operates as a 
5 pointing device to invoke other functions of the computer, such as editing text contained in the 
display field, and changing the insertion point in the display field. 

Typical prior methods of data entry with a stylus present the following difficulties 

to the user. 

10 1) visual attention must constantly be shifted between the entry and display 

fields; 

2) the stylus must be moved repeatedly between the display fields, to perform 
editing functions, and the entry fields, to continue entering handwritten data; 

3) the separate entry fields may use as much as one half of the available 
15 graphics display area on a small hand-held device such as a PDA, reducing the amount of other 
information that can be displayed; 

4) often, users must select the desired writing mode (characters, numbers, 
punctuation) and may forget which writing mode is currently active, or may enter the wrong type 
of handwritten symbol in an entry field; and 
20 5) in many systems each entry field accepts a single character only, which must 

be recognized before the system will accept further handwritten data. 

U.S. Patent No. 5,528,743 issued to Tou et al.describes a method of inputting 
information for the purpose of word processing. In word processing displayed text is formatted 
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into words, sentences ana paragraphs. Tou et al. describes a number of calculations based on the 
spatial relationship of the position of newly entered graphical input symbols to the p ositions nf 
paragraphs of formatted character data already existing on the display. Displayed character data 
are formatted into paragraphs, not necessarily superimposed on input symbols. For example, in 
5 Figure 3a of Tou et al. the handwritten symbol "three" entered near the right margin is displayed 
in Fig. 3b near the center of the line. The handwritten input below the existing character data in 
Fig. 3b is displayed as character data on the line above Fig. 3c, not in the position in which it was 
entered. 



10 European Patent Publication No. EP 0689124 (Canon) is concerned with 

recognizing handwritten symbols which are gestures and distinguishing these from other 
handwritten input, and performing editing and other operations specified by the gestures. The 
interaction described in the body of the patent, and in their claims, requires several steps with 
menu prompts for the user to execute a gesture action. In the 'Description of Related Art' Canon 

15 states that when handwritten characters are input to arbitarv positions, recognition processing for 
the characters is performed and the recognition results are displayed at the positions where the 
handwriting input operation was performed. However, it is not disclosed whether a user must 
wait after writing each character until the system analyses and prints out that character or carries 
out the command operation represented by that character before writing the next character. It is 

20 also not disclosed whether or not the recognized input characters are superimposed on the 
handwritten input characters or are merely placed near them. 
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U.S. Patent No. 5276794 issued to Lamb describes the handwritten entry into a 



single-line field based on the position where the handwriting is initiated, or in the field that is 
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active, even if some or an of the handwriting falls outside the display field. Chara cter data are 
displayed left-justified in the single-line field, regardless of location of the handwritten inp ut 
For example, in Fig. 2a the four symbols "FRED" are entered in the right half of the inp ut fie1ri n 
but the recognition is displayed near the margin (see Fig. 2b). In addition, the recoenition and 
5 display data are not immediate, but triggered by a time-out or other strategy to determine when 
the user has completed the input for a field. 

U.S. Patent No. 5,220,649 issue to Forcier describes a system in which 
handwritten input is displayed as "digital ink" (i.e., not automatically recognized and converted 
10 to character data) and editing mode is initiated with a timeout. 

Accordingly, it is an object of the present invention to provide an improved means 
of data entry and editing by superimposing the input field and the display field on a GUI. It is a 
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further object of the invention to provide an interface in which graphic symbols are entered by 
the user in an input field, and then are immediately replaced with the symbols' corresponding 
character data in. approximately the same location. It is yet a further object of the invention to 
provide a means of correcting and editing character data without moving the stylus outside the 
5 input field. 

SUMMARY OF THE INVENTION 

According to the invention there is provided a pen or stylus-operableted system 
graphical user interface for a computer or computing device, which includes a graphical user 

10 interface coupled to said computer and having a sensing surface , the sensing surface controlled 
by the graphical user interface and having means an area corresponding to a data input field, the 
data input field being conditioned for hand entering and editing of graphical input symbols.t-and 
[h]Handwriting recognition software on the computer is operative to analyze the graphical input 
symbols one after another without interruption or delay and to superimpose a display field of 

15 character data corresponding to the graphical input symbols on the data input field. 

Advantageously, the sensing surface is a display surface. Alternatively, the 
sensing surface could be a tablet separate from the display surface. 

20 The handwriting recognition software also initiates an action based upon the 

graphical input symbol. Preferably, the action is an editing mode wherein the pen or stylus 
contacts the sensing surface without moving for a predetermined minimum amount of time. 
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Symbol recognition of handwritten input is a default mode and editing mode is 



initiated with a timeout. 

5 

Preferably movement of the pen, in predefined ways, without being removed from 
data input field, causes corresponding editing functions to be effected. 



The character data may be corrected and edited in the editing mode without 
10 moving a cursor for the pen or stylus outside the data input field of the sensing surface. 

In another aspect of the invention there is provided a method of a pparatus for 
combining data entry of handwritten symbols with displayed character data in a pen or stylus- 
operablet ed graphical user interface for a computer or computing device, which includes means 
15 for recording and displaying handwritten graphical input symbols as they are entered on a data 
input field of a display surface as they are entered ; and handwriting recognition software for 
analysing continuously, without interruption or delay and automaticall y the graphical input 
symbols with handwriting recognition software and superimposing on the display field character 
data corresponding to the graphical input symbols. 

20 

Preferably, the means for recording is a sensing surface operative to receive and 
record the graphical input symbols are entered on a sensing device . The means for displaying is 
the sensing device may be separate from the display surface or, alternatively may be a part of 
the display surface. 
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The handwriting recognition software may initiate an action based upon the 
graphical input symbol. The action may be an editing mode when the pen or stylus contacts the 
display surface for a predetermined minimum time without moving. 

5 

Movement of the pen in predefined ways, without being removed from the data 
input field, may cause corresponding editing functions to be effected. 
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c^^ata may be corrected and edited in the 



Characteraata may be corrected and edited in the editing mode without moving 
the pen or stylus outside the data input field- 
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I CLAIM: 

1 . A pen or stylus operated operable system for a computer or computing device , 
comprising: 



(a) a graphical user interface coupled to said computer (1 10. 1 12, 1 14) and having 
a sensing surface (11) said sensing surface having an area corresponding to a 
data input field (13) , said sensing surface conditioned controlled by said 
graphical user interface and having means for hand entering and editing of 
graphical input symbols (31, 32) : 

(c) handwriting use^recognition software on said computer operative to analyze 
said graphical input symbols (31, 32) one after another without interruption or 
delay and to automatically superimpose on and replace said graphical input 
symbols with a display field of character data (33) corresponding to said 
graphical input symbols (31, 32) on said data input field (13) . 

2. An interface according to claim 1, wherein said sensing surface (112) is a display 
surface. 

3. An interface according to claim 1, wherein said sensing surface (112) is a tablet (11) 
separate from a display surface. 

4. An interface according to claim 1, wherein said handwriting recognition software also 
initiates an action based upon said graphical input symbol. 
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5. An interfac^roording to claim 1, wherein said handwritin^recognition software 
initiates an editing mode when said pen or stylus (12) contacts said sensing surface (11) 
without moving for a predetermined minimum amount of time. 



6. An interface according to claim 5, wherein said symbol recognition of handwritten 
input is a default mode and editing mode is initiated with a timeout minimum amount of time 
is 200 msec . 

7. An interface according to claim 5, wherein movement of said pen (12) , in predefined 
ways, while being held in continuous contact with the sensing surface, without being 
removed from said data input field, activates causes corresponding editing functions to be 
effected . 

8. An interface according to claim 7, wherein said character data is corrected and edited 
in said editing mode without moving a cursor for said pen or stylus (12) outside said data 
input field of said sensing surface (11) . 

9. A pparatus for method of combining data entry of handwritten symbols with displayed 
character data in a pen or stylus-operableted graphical user interface for a computer or 
computing device, comprising: 

(a) means for recording and displaying handwritten graphical input symbols as 
they are entered graphical input symbols on a data input field of a display 
surfac e as thoy are entered ; and 
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(b) userl^bgnition software for analyzing said graphic^nput symbols one after 
another continuously without interruption or delay with handwriting 
recognition software and automatically superimposing on and replacing said 
graphical input symbols (3h 33) with a display field character data 
corresponding to said graphical input symbols. 

10. A pparatus A method according to claim 9, wherein said means for recording is a 
sensing surface (11) operative to receive and record the graphical input symbols are entered 
on a sensing surface . 

1 1 . A pparatus A method according to claim 10, wherein said means for displaying 
sensing surface is separate from is said display surface (112) . 

12. A pparatus A method according to claim 10, wherein said sensing surface (112) is at 
least part of said display surface (112) . 

13. A pparatus A m e thod according to claim 9, wherein said handwriting ttseMecognition 
software also initiates an action based upon said graphical input symbol. 

14. Apparatus A method according to claim 9, wherein said handwriting user-recognition 
software initiates an editing mode when said pen or stylus (12) contacts said display surface 
(112) for a predetermined minimum time without moving. 



15 

Mark-Up Sheets 



15. 



A pparatus 




according to claim 14, wherein move! 




it of said pen, without 



being removed from said data input field, in predefined ways, while held in continuous 
contact with the sensing surface, activates causes corresponding editing functions to be 
effected. 

16. A pparatus A method according to claim 15, wherein character data is corrected and 
edited in said editing mode without moving a cursor for said pen or stylus outside said data 
input field. 
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